FIG. 1 shows a system for viewing digital video while listening to its corresponding audio. The system includes a digital video display 104 and one or more speakers 102. The digital video display 104 is driven by display driving circuitry 103; and, the one or more speakers 102 are driven by speaker driving circuitry 101. The display driving circuitry 103 receives a stream of video “samples” and renders them onto the display 104. The speaker driving circuitry 101 receives a stream of audio samples and causes the speaker(s) 102 to produce corresponding audio sounds. The term “samples” is a historical term that refers to the digital capture of video/audio information from its naturally analog form. The term samples should nevertheless be understood to also refer to video and/or audio information having a non-analog source (e.g., computer graphics, digitally synthesized music, etc.).
FIGS. 2a through 2c relate to differing aspects of the manner in which a video sample stream may be formatted. Specifically, FIG. 2a relates to different ways in which a video sample may be encoded; FIG. 2b relates to different ways in which the video samples may be ordered within the video sample stream; and, FIG. 2c relates to differing amounts of video samples that may exist in the video sample stream per rendered horizontal line of displayed imagery and per rendered vertical line of displayed imagery.
FIG. 2a shows three different video sample encoding formats 201, 202, 203. Each of the video sample steams 201, 202, 203 show their corresponding encoding approach across seven samples 200 (S1 through S7). Video sample stream 201 corresponds to a “trichromatic” encoding approach. Both of video sample streams 202 and 203 correspond to “color difference” encoding approaches.
According to a “trichromatic” encoding approach, each video sample contains three color components where the mixing of any two color components cannot produce the third color component. Video sample stream 201 provides an example of the most commonly used trichromatic approach “RGB”. RGB breaks down each sample into red (R), green (G) and blue (B) components. Thus, the first sample S1 in video sample stream 201 corresponds to the red, green and blue components (R1G1B1) of the first sample's visual information, the second sample S2 in video sample stream 202 corresponds to the red, green and blue components (R2G2B2) of the second sample of visual information, etc.
Both of video sample streams 202 and 203 employ a “color difference” encoding scheme. Color difference schemes use a luminance related quantity called luma (e.g., Y) and a pair of color difference components called chroma (e.g., Cb,Cr). Color difference schemes are often characterized by the sample rate relationship as between the luma and chroma components. According to a first type, referred to as 4:4:4, the luma and chroma components have the same sample rate. Thus, as seen in the 4:4:4 color difference sample stream 202, both luma and chroma components exist for each sample in the sample stream. According to a second type, referred to as 4:2:2, the chroma components have half the sample rate as the luma component. Thus, as seen in the 4:2:2 color difference sample stream 203, each sample has a luma component but only every other sample has chroma components.
FIG. 2b relates to different ways in which the video samples may be ordered within the video sample stream. A first is referred to as “progressive” and a second is referred to as “interlaced”. According to a “progressive” approach the samples within a video sample stream are ordered so that each of the frames of the video, when presented on the display, correspond to a complete “full screen” picture. FIG. 2b attempts to depict a progressive display approach by showing a series of frames 210 through 213 each having a complete set of horizintal “scan” lines of displayed samples.
By contrast, according to an interlaced approach, the samples within a video stream are ordered so that the frames of the video are presented on the display as a pair consecutive “fields” where each neighboring horizontal “scan” line in a field corresponds to every other horizontal “scan” line on the display. FIG. 2b attempts to depict an interlaced display approach by showing a sequence of fields 214, 215, 216 and 217, where fields 214 and 216 are located only on even numbered scan lines, and fields 215 and 217 are located only on odd numbered scan lines.
FIG. 2c demonstrates that different video sample streams may have different amounts of video samples per rendered horizontal line of displayed imagery and per rendered vertical line of displayed imagery. The samples, or picture elements in this rectangular array are commonly referred to as “pixels”. Here, the video sample stream should be formatted so that its samples correspond to (or at least can be rendered upon) the display upon which they are to be shown. Many video display types are designed to render imagery across an arrangement of pixels. The arrangement of pixels is typically characterized as the number of pixels that extend horizontally across the display and the number of pixels that extent vertically across the display.
FIG. 2c shows a representation of the pixel layout for two different video displays 218, 219. The first video display 218 has a pixel layout of 480 vertical pixels by 640 horizontal pixels. The second video display 219 has a pixel layout of 1080 vertical pixels by 1920 horizontal pixels. Note that different aspect ratios can be achieved based upon the pixel layout. That is, the first video display has a 4:3 horizontal to vertical aspect ratio (i.e., 640/480=4/3) that is typical of standard television; and, the second video display has a 16:9 horizontal to vertical aspect ratio (i.e., 1920/1080=16/9) that is more typical of standard cinema.
Because a display's pixel layout can be viewed as an array, a display can be viewed as having Y number of horizontal lines that each have X pixels; where, Y is the vertical pixel count and X is the horizontal pixel count (i.e., display 218 can be viewed as having 480 horizontal lines that each have 640 pixels; and, display 219 can be viewed as having 1080 horizontal lines that each have 1920 pixels). Here, referring briefly back to FIG. 1, for a video sample stream that is provided to a display's driving circuitry 103, the number of samples per horizontal line in the sample stream ideally correspond to the number of pixels per horizontal line in the display 104; and, the number of horizontal lines' worth of samples per frame in the sample stream ideally corresponds to the vertical pixel count of the display 104.
Because of the various ways in which a video sample stream may be formatted, specific signal processing blocks have been used to convert from one type of formatting to another type of formatting. FIGS. 3a through 3d demonstrate some of these signal processing blocks.
FIG. 3a shows a color space converter 301 (CSC). The color space converter 301 is responsible for converting from one type of sample encoding to another type of sample encoding. For example, color space converter 301 might convert trichromatic encoded samples into color difference encoded samples; or, color space conveter CSC could covert color difference encoded samples into trichromatic encoded samples. FIG. 3b shows a pair of converters 302, 303 for converting between different forms of color difference conversion. Specifically, converter 302 converts 4:4:4 color difference encoded samples into 4:2:2 encoded samples; and, converter 303 converts 4:2:2 color difference encoded samples into 4:4:4 encoded samples.
FIG. 3c shows a deinterlacer 304. Deinterlacer 304 converts an interlaced video sample stream into a progressive video sample stream. Various types of deinterlacing schemes exist. A basic approach involves simply packing the content of a pair of consecutive fields into a single progressive frame. However, this approach can render noticeable defects for fast motion images if neighboring fields correspond to different moments of time (i.e., an object of fast motion will blur). As such, “motion adaptive” deinterlacers have been developed that intelligently calculate the missing content between neighboring field lines in order to produce a complete progressive frame.
Another type of deinterlacing technique, referred to as “inverse 2-3 pulldown”, effectively reverses a “2-3 pulldown” interlacing process. The 2-3 pulldown process converts frames from a 24 frame per second stream into a 2-3-2-3- . . . field pattern (e.g., a first frame is used to create a pair of fields, the next frame is used to create a trio of fields, etc.) so as to create a 60 field per second stream. Inverse 2-3 pulldown combines fields from the same original film frame to form a complete video frame. Because fields are combined by this process so as to create frames, “inverse 2-3 pulldown” is viewed as a deinterlacing technique. Another type of deinterlacing technique, referred to as “inverse 2-2 pulldown”, effectively reverses a “2-2 pulldown” technique that uses each frame to create a pair of fields.
FIG. 3d shows a frame rate converter. A frame rate converter changes the formatting of a sample stream from being compatible with a first frame rate (e.g., 60 frames per second (fps)) to being compatible with a second frame rate (e.g., 30 frames per second (fps)). Frame rate converters are typically constructed to provide an output stream that, when viewed, exhibits rendered images having the same speed of motion that the input stream exhibits at its corresponding frame rate. That is, typically, the frame rate conversion process is not supposed to change the speed at which moving images move when displayed. A frame rate converter typically accepts input information that informs the frame rate converter of the specific conversion that it is to take place (e.g., a specific rate upconversion or rate downconversion)
FIG. 3e shows a scaler 306. A scaler converts a video sample stream that is formatted for a first pixel array into a sample stream that is formatted for a second pixel array. For example, referring also to FIG. 2c, a scaler 305 would be used to convert a sample stream that is formatted for the 1920×1080 display 219 into a sample stream that is formatted for the 640×480 display 218 (e.g., by intelligently “dropping” certain samples). Likewise, a scaler 305 could also be used to covert a video sample stream that is formatted for the 640×480 display 218 into a sample stream that is formatted for the 1920×1080 display (e.g., by determining appropriate pixel spacings between those pixels that will be illuminated with sample data from the 640×480 compatible stream). Other than dropping pixels (when scaling from a higher pixel density display to a lower pixel density display) or repeating pixels (when scaling from a lower density pixel display to a higher pixel density display), more advanced methods such as finite impulse response (FIR) filtering or linear interpolation may be used. A scaler typically accepts some form of input information (e.g., a scale factor) that informs the scaler of the specific scaling that it is to perform.
Note that any of the signal processing blocks of FIGS. 3a through 3e may be implemented in whole or in part with dedicated hardware (e.g., CMOS circuitry or programmable logic designed to perform the signal processing functions) or in software (e.g., a general purpose or digital signal processor that executes instructions to perform the signal processing functions).